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This paper discusses the effects, of an evaluability 
assessment, rapid feedback assessment and performance 
monitoring system on the design, management and 
modification of Federal special education programs. 

The purposes and methodology of the assessment process 
will be presented^ followed by a discussion of the ways in 
which the assessment techinques have been used successfully 
to meet the needs of policymakers and program mangers in 
the U.S. Department of Education's Special Education 
Programs- (SEP) ♦ Lastly, the ways in which the assessment 
information h^s been used by, SEP to describe, analyze and 
change policy and programming will bo^r discussed . 

r 

The evaluability assessment (EA) , rapid feedback 
assessment (RFA) and performance monitoring system 
discussed in this paper were conducted under an umbrella 
contract with the U.S. Department of Education's Division 
of Performance Management Systems. They are three of the 
fifteen short-term assessments conducted by the American 
Institutes for Research; (AIR) ginGe 1980. 

/ 
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THE ASSESSMENT PROCESS " 

t 

Evaluability assessment is a' management tool used to 
provide administrators and policymakers with immediately 
useful information on the effectiveness of their programs. 

4 

The assessment process, as adapted by AIR/ consists of 
three distinct phases: the evaluability assessment/ the 
rapid feedback assessment/ and the performance monitoring 
system. These phases are sequential: the design of a- 
performance monitoring system generally follows a rapid 
feedback assessment/ and a rapid feedback assessment 
follows an evaluability assessment. Yet/ it is the 
'information needs of the program managers and policymakers 
that determine "the extent of the work to be conducted/ and 
there are many times/ for example/ when an EA or an EA/RFA 
may be sufficient. 

4 

Evaluability Assessment 

r 

evAuability assessment * is used to determine the 
extent to which a program can meet its goals/ and the 
extent to which it is, or is not ready fo? further 
evaluation. 



An evaluability assessment is generally a five-stage 
process. The five stages serve to answer the following 
questions ; 

What is the program supposed to look like?. 

Through a series of interviews with program management 

c 

and policymakers (where appropriate),, and a review of 
internal and external program documents, the assessment 
team developes a description of the program. Several types^ 
of descriptive program models are then developed tha* 
depict: * . • 

• the intended logic of the program; and 

• the activities and proce^sses used carry 
out this logic. 

The program description and the logical models 
document the extent to which th^e agreed-upon program 
objectives, measures, and measurement systems are in place, 
and the extent of agreement or disagreement among the 
different perspectives represented in the models. 

What does t>he program actually underway look like? 

In this phase, the description of'the intended program 
is compared to the way the program actual ly.,.>oPjeLr.a±.es^^^n 



order to obtain, an accurate understanding of activities in 



the field/ staff conduct site visits to a small sample of 
program grantees and contractors.. While on site^ 
interviews are conducted y/ith, grant administrators and. 
service providers, and documentation describing project 
goals and activities is collected. In addition to 
providing a clear picture of grantee activities, site 



visits are used to: 



determine the type and availability of 
grantee performance data; 

assess the measurabi li ty of program 
objectives at the field level; 

obtain field staff views on the 
feasibility of the program's strategy for 
success; and 

identify discrepancies between the 
intended program and the actual program. 



This site specific information is used to generate 
function models de^picting the flow of activities, 
information and resources that result from the Federal 
program. 

Which objectives are plausible given the program as it is 
currently - operating? 

Based on all the information gathered, and models 



developed to that. point, the plausibility of each 
previously defined. program objective is assessed. The 
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discrepancies identified in the previous stage are examined 
to determine whether or not it is reasonable to' expect that 
the program will accomplish it3 objectives. . 



What are possible measures or indicators of program 
performance? 

The plausibility information is then used to develop a 
model identifying performance measures that can be used to 
assess the program's progress toward accomplishin^its 
objectives. Potential data sources that can provide th^ 
^measurement information are also identified. If resources 
were infinite^ a complete evaluation would be able to 
assess every activity and outcome. As resources are often 
limited/ only those program components of most importance 
to program administrators and staff are measured. 

The me'asurement models are examined for their utility 
and accuracy/ and those measurement issues and program 
performance questions of the highest priority for further 
evaluation are identified. 
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What management and evaluation, options can be undertaken 
for program improvement? 

Finally, assessment staff recommend a series of 
specific management and evaluation actions that the program 
can implement immediately, to iH[nprove its functioning . 
Reviewed by program staff, whose comments are incorporated, 
these actions often propose strategies to reduce or 
eliminate the discrepancies identiifed earlier. 

Rapid Feedback Assessment 

An RFA builds upon the information gathered during an 

EA. 

To summarize the previous section, an EA determines, 
in part: (J.) the extent to which a program is ready for,- 
evaluation; (2) what changes can be made to make the 
program more manageable and accountable; and (3) what 
measurement issues and program performance questions could 
be investigated in later evaluation activities. 



An RFA focuses on those components of a program found 

k 

in the EA to be ready for evaluation and: 

• provides a preliminary evaluation of 
program performance by summarizing 
available /or readily obtainable 
information on the efficiency. 
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ef f ectiveness / and responsiveness of the 
program; 

• identifies indications for change in 
program activitites that would enhance 
program performance; 

• analyses and suggests solutions to the 
problems likely to b'e encountered in 
further full-scale evaluations. 



The scope of an RFA is defined during meetings with 
program management. The evaluation and performance 
questions identified in the EA provide the basis for a set 
of questions to be answered^ by an RFA. The final set of ^ 
questions to be answered by the RFA are based on 
(1) program management's specific needs for immediately 
usable information and (2) the feasibility of answering the 
questions meaningfully within the time frame of the , RFA. 
For each question, one or more standards are developed by 
which the adequacy of the information gathered can be 
assessed. * ^ 

The answers to the agreed upon list of evaluation 
questions constitute the major body of information 



generated by a rapid 
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The program performanc 
RPA pinpoints areas of progr 
disfunction. We have notices 



that often program 



disfunction is caused by a combination" of: activities that 
ai(e not occuring as planned or needed; and by the lack of 
clear and reliable program* performance data. * 

• J 

To remedy this, we have/ added a third .phase to the 
e^luability/ rapid feedback/ asssessment process : the 
design of a monitoring syst/em designed to allow program 
managers to.^evaluate the performance of their program. The 
system uses graphic models to describe the activities that 
constitute an effective program, and then presents the 
evaluation questions that need to be asked about the* 
activites&in order to ^onitor program effectiveness. 
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.MULTI*-PHASED ASSESSMENT OF . 
THE CAPTIONED FILMS* LOAN SERVICE FOR THE DEAF ' 

A multi-phased assessment of the Captioned Films Loan 
Service for the Deaf was conducted from jJ.artuary to November 
of 1982. The Captioned Films Loan Service for the Deaf is 
a free loan service to ^the deaf of captioned educational 
and theatrical (general-interest) films. The films are 
intended for the non-profit use of deaf persons^ teachers 
of the deaf/ interpretors ^ parents, and guests. 

The Captioned Films Program responsibilities include:"* 
film selection (for captiofiihg) / acquisition, captioning, 
distribution, and management of the loan servic/e, ^ 

During the first five months of the year, we conducted 
an evajuabilit^ assessment of the Cap1;ioned Films Program. 
The EA served to promote consensus on the program goals and 
determine: (1) the extent to which the program would be • 
able to meet its goals, and (2) the extent to which the 
program was mesujrable and evaluable. This first phase of, 
the assessment also identified programmatic issues and 
concerns needing further examination. 

Between May and September, we conducitSd a rapid 
feedback assesma^ of the Captioned Films Program. .-This , 



phase provided up-to-date, usable program performance 
information in responsqf^o those questions and concerns 
raised during the evaluability assessment. 

The^-final phase of the assessment process (October and 
November 1935) involved the development of an integrated 
performance monitoring system. The system was designed to 
help sustain the improvements ^n program efficiency and 
effectiveness that had resulted from the previous phases of 
the assessment. 

4 

^Context of the Assessments , 

The assessmentS/of the Caption^d-^ Films Loan Service 
fo^r^he Deaf wer^ conducted at the .request of the Director 
of the Educational Services ^ivisii^ (Special Education 
Programs) in whose division the program was housed. At the 
time o-f t^e assessment, the Captioned Films Program had 
been in operation for over twenty years. While the scope 
of the program had grown since its inception in 19^, the 
basic structure of the film delivery service had remained 
unchanged. Facing potential budget cuts, the program was^ 
becoming less able to meet the increased demand for 
captioned educational (for schools) and theatrical (for^ 
film clubs) films. 
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The program' s main functions of film selection and 
distribution were managed through one contractor. That 
contractor was, in -turn, responsible for administering the 
library from which the theatrical captioned films (TCF) 
were di str ibuted , and for administering the subcontracts to 
the^eo depositories from which the educational captioned 
films (ECF) were dis.tributed . 'While it was not involved 
with day-to-day program operations, the Federal program 
office had maintained, in addition to monitoring and 
reviewing, th^ major function of film acquisition. 

Summary of the Evaluability Assessment Findings 

' ' r ■ . 

The long-term goal of the Captioned Films Program is 

to promote the-^eneral welfare of the deaf by providing 

films that enrich tfieir educational and cultural 
r • * 

experiences, and bring them into better touch with the 
realities of their environment. 

The evaluability assessment identified seven immediate 
program objectives on which achievement of the program's 
long-range goals must ultimately^ rest. These objectives 
are : 

1. Films are evaluated* and selected in a way 
that reflects and responds to consumer . 
needs; 

11 14 




2. Film captions and study guides doffe 
in a timely and cost-effective manner; 

3. Film prints are acquired in a 

^ cost-effective and efficient manner; 

4. Consumers of educational captioned films 
(ECFs) and theatrical captioned films 
(TCFs) are aware of available films; 

5. ECF and TCF consumer registries are 
developed and expanded to include all 
eligible users; 

6. Films are booked and shipped in the mos^: 
effective and timely manner; 

7. Consumer evaluations of the films and the 
service are used to improve program 
functioning . ^ 

Assessing the Captioned Films Program's readiness for 
evaluation entailed judging whether these agreed-upon 
objectives were plausible and measurable and v^hether 
program performance information was available to indicate 
progress toward achieving the objectives. 

\ Findings from the evaluability assessment relating to 
each of these points are summarized, below. 

Plausibility of Objectives 

The results of the plausibility analysis of the 
Captioned Films Program's immediate objectives were mixed* 
Briefly, three of the seven objectives (Numbers 3/ 4, and 
^7) were judged to be entirely plausible, none was found 
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implausible, a^nd the remaining four (1, 2, 5, and 6) were 
plausible in part. The mechanisms to support the success 
of the Captioned Films Loan Service for the Deaf appeared 
to be in place and to be functioning adequately, if not 
perf^tly,, 

Measurability of Objectives and Feasibility of Locating 
Usable Sources of Performance Data 

The evaluability assessment next assessed the 
measurability of the seven immediate program objectives 
discussed above and determined the utility to program 
management of acquiring further information regarding each 
objective. Working with program management, we identified 
three evaluation concerns that were of interest and that 
were amenable to measurement: 

• How adeqoate are the EXF collection and 
services for meeting users* needs? 

How adequate are the TCF collection and 
services f^r meeting users* needs? 

How cost effective are the film 

acquisition and distribution procedures? • » 

We concluded that these evaluation concerns were measurable 
and that data relating to program measures were for the 
most part available or easliy obtainable. 
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' Summary of the Rapid Feedback Assessment Findings 

A set of evaluation questions was developed for the 
three areas of concern identified during the evaluability 
assessment. Responses to these evaluation questions 
constituted the main body o^ information generated during 
the rapid feedback assessment . The responses to the 
questions are summarized below; 

Adequacy and Effectiveness of 
ECF Collection and Service" 

The rapid feedback assessment revealed that the ECF 
distribution system functioned with a high degree of 
efficiency (with better than a 95% success rate in booking 
requested films) . The implementation of a computerized 
system of sharing films among depositories (FILMSHARE) in 
1980 enabled the vast majority ofkccounts to receive the 
films they want on the dates they^want "all or most of the 
time." As funds no longer must be spend making a copy of 
every captioned film for each depository, more new titles 
can be added to the ECF collection than was previously 
possible. 

The process used to evaluate and select educational 
films to be captioned raised more questions than did the 
distribution system. Two major concerns arose: (1) much 
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of the process was undocumented, and therefore impossible 
to assess with any .certainty; am3 (2) the varying ways in' 
which data were collected and reported made it extremely 
difficult to assess performance at each step of the 
process: from needs assessment, through issuance of 
curriculum priorities, to the final selection of films. 



Adequacy and Effectiveness of 
TCP Collection and Service 



The procedures for nominating, evaluating, and 
selecting theatrical films for captioning appeared to be 
working effectively. However, there was no provision for 
obtaining input on reactions from users, so it wa s^no^t 
possible to assess the extent to which user's needs were 
being met. 

There was considerable evidence to suggest that users 
wer^Pftot receiving the films they most want. There also 
appeared to be some discrepancy between the way the booking 
procedure was described and actual practice, with the 
result that on at least some occasions films appear to have 
been booked to users that the users had not included their 
most recent list of 40 requests. Because data on 
individual account holder's requests and bookings were not 
maintained in a readily analyzable form, it was rfot 
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possible to determine precisely whether and to what extent 
this overbooking in fact occured. 

Inventory levels were inadequate to meet requests for 
the more popular films, as the number of requests exceeded 
the 'estimated maximum number of bookings possible for over 
50% of the films in that caT^ory. For all but one of 
these "more popular" films, only 12 prints were available^ 
suggesting that many of them were new additions to the 
collection and that the need for additional film prints for 
those titles had not been assessed. '* 

Relatively little information on TCF account holders 
was collected or maintained; what was available (i.e./ 
organization type and loca^tion by state) ^was reported as 
part of the combined ECF/TCF account-holder data. 
Information on orders, bookings, and audience count by user 
category was compiled semiannually to proyide a basis for 
monitoring TCF usage. However, there was no provision for 
systematically obtaining feedback from users. 

Cost-Effectiveness of the Film Acquisition 
and Distribution Procedures 

The rapid feedback assessment revealed that the cost 
effectiveness of. the film acquisition and distribution 
procedures differed for the ECF and the TCF collections. 




Costs for development of the ECF collection and 
shipment of these films to clients were reasonably well 
documented and appeared to be satisfactory both to film 
users and the Federal office. 

Fewer and less reliable data were available for the 
costs associated with the TCF collection. Film 'development 
and captioning costs were not sufficiently well doucmented 
to determine just what actual costs are. TCF shipping 
costs were documented. 



The EA/RFA process is an iterative one relying on 
frequent contact between the evaluator and the client. We 
met with program managers and staff about once a month. At 
these meetings/ all findings to date were reviewed and 
discussed. * 

The discussion process allowed program administrators 
to think about the Captioned Films Loan Service for the 
Deaf in ways that day-to-day program management rarely 
leaves time for. For examprfe/ staff and management' 
together developed a wjorlcable statement of program intent 
and objectives; program management received and reviewed 
up-to-date information on how the program actually 



Use of the EA/RFA Findings 



functioned in the field, and the extent to which they would 
be able to meet their objectives; and they reviewed and 
agreed to the performance and measurement issues to be 
addressed in the RFA. Since the process and the findings 
were discussed along the way, SEP had 'already begun to 
consider certain ^^rogram modifications (i.e., dismantling 
the Handicapped Learner Materials component of the 
Captioned Films Program, which took up storage space, and* 
for which there was almost no demand) by the time the RFA 
Final Report was submitted. The EA/RFA information was 
used 1^ modify and improve program activities during the 



The RFA Final Report coincided with, and was used as 
the basis of the new procurements for the Captioned Films 
Loan Service for the Deaf. Our recommendations for further 
' program improvements and special investigations to be 
carried out by the contrNSctors or a third-party evaluator, 
were incorporated into the new procurements. 

While these specific recommendations went some way 
toward improving program functioning and internal ^ 
monitoring, it had become evident during the' RFA that the 
way in which program data were organized and analyzed 
hindered effective monitoring of major elements of ,the 



course of the assessments. 
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Captioned Films Loan Service for the Deaf. The Captioned 
Films Loan Service needed to implement a well-designed 
performance monitoring system. --^ 

Summary of the Performance Monitoring System 
* 

^ The program , performance information generated by the 

rapid feedback assessment and summarized in the previous 

sections pinpointed areas of program function and program 

disfunction. Several areas were noted for which clear and 

reliable information program performance was not readily 

available. A third task was undertaken: the design of an 

up-graded performance monitoring system, 
f 

The purpose of the effort was twofold. We suggested 
changes in and/or modifications to certain activities that 
would improve program performance. We also designed a 
monitoring system that would encourage the regular 
collection and reporting of information on all key aspects 
of.^progtram performance . 

> 



The first step in designing this performance 
monitoring system was to develop graphic models of the 
program to clarify the functions of the actors involved in 
the Ca^ptioned Films Program. Whil§ the models generally 
represented the program as it operated, new activities and 



njodif ications to procedures suggested by the results of the 
assessments were included where appropriate. The lecond 
step was to specify the evaluation questipns we believed 
needed to be asked in order to evaluate the ongoing 
performance of each phase of ECF and TCP operations — 
selection, captioning, and distribution (which includes 
promotion, booking and shipping). 

The third step entailed the actual design of the 
monitoring system. This system, presented as a series of 
"performance monitoring charts," detailed the data 
collection and reporting activities that needed to occur to 
answer the evaluation questions introduced in the program 
models . 



The performance monitoring system proposed in this 
report was a detailed and comprehensive one that could be 
implemented either in whole or in part, dependin^on 
program managers' prioritie's ancK^resojarces . The 
modifications and innovations suggested dealt with all 
aspects of the Captioned Films Program. These 
recommendations that we believed should take priority are 
summarized below. 
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Highlights of Recommendations for Educational Captioned 



Fi 1ms 



• Jfeview all pre-selected films in the field 

(rather than reviewing some in the field 
and some in an , evaluation workshop); 
reserve the annual workshop for 
"validation" of all field recommendations. 

• Categorize films by curriculum priority 
areas throughout the evaluation and 
selection process. 

• Shift responsibility for negotiating the, 
purchase of films and the development of 
spotting lists from CFMA to the 
contractor . 

• Replace the bound volumes of studyguides 
with shorter, loose-leaf guides to be 
distributed to users with the films. 

• Redesign the audience response form to tap 
only areas of greatest concern and use 
this feedback as a systematic and ongoing 

. user needs assessment. 

• Keep records of requests and bookings by 
individual account number ; periodically 
identify inactive accounts and take steps 
to reactivate or replace them. 

• Monitor film usage by curriculum area and 
by interest level to shape the collection 
more closely to user demand. 

• Develop a standard BICS procedure manual 
for use by depositories. 




Highlights of Recommendations for Theatrical Captioned 



Films 



• Subject film nominations from all sources 
to systematic review and evaluation prior 
to selection. 

• Document the criteria used in nominating^ 
evaluating, and selecting theatrical 
films. 



Bring TCF booking practices into line with 
the procedures described by the contractor 
and document the booking decisions to 
explain discrepancies between requests and 
bookings • 

Redesign shipping forms to collect and 
assess audience response to the films and 
the service. 

Ke6p records of requests and bookings by 
individual account holder numbers and 
according to uniform categorization of 
user groups. Develop a standard by which 
to judge the adequacy of film inventories 
for meeting user demand. 

Document all time and dollar costs related 
to present shipping methods ' to identify 
methods that will satisfy users, while 
allowing maximum circulation of the" TCF 
collection. 



A summary listing of the reports and documents 
suggested by the performance monitoring system is presented 
below. We suggested that these documents need not 
necessarily be Separate, formal reports, but could be 
sections in an annual report, manuals, informal 
compilations of data, or annotated computer printouts.. The 
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suggested reports were intended to present a cohesive 
picture of. program operations to program management for 
their review and considerition. 



ECF 



Needs assessment report 
Film evaluation report 
Film selection report 
Captioning report ; 
Outreach and promotion report 
Procedural manuals (FILMSHARE & BICS) 
Usagef^-areport ' 
Shipping report 
Damage report 



TCF 



■7^ 



Feedback report on audience response/ 
shipping form 

Film nomination and evaluation report 

Film selection report'^ 

Captioning report 

Account registration and activity 
report 

Booking report 

Inventory report * «^ 
Shipping report ' 
Loss and damage report 



The proposed performance monitoring system was 

intended to avoid imposing undue time or cos1: burdens on 

• 

the Captioned Films Program or its contractors. To the 
greatest extent possible, the system relVed on then, current 
procedures and already collected, data . Most of, the changes 
recommended pertained to organizing and examining these ^ 
data in new dr different ways. .Much of the new data' 
requested was already in the pro^am's various compViter 
information systems — the^CB^poking system, FILMSHARE, 
and BICS. In some cases, new compu^r programs would have 



• \ 

to be written in order to retrieve the information in the 
format suggested by the performance monitoring system. In 
the case of film requests by TCP users, new data would need 
to be added to existing Computerized information systems. 
However, greater reliance^ on computerized record systems 

c 

would streamline the process of data summary and reporting. 

Examples of the prototype models and performance 
monitoring charts for ECF selection ^nd captioning are 
presente(^in Appendix A, 

Use of the Performance Monitoring System 

]/ In FY-83, the Captioned Films Loan Service for the 
Deaf mov?d from its longtime location to a new division 
within SEP. While the branch chief responsible for the 
overall program management moved with the program, the new 
division director and program staff had not had the benefit 
of ongoing involvement in the EA/RFA process. The validity 
of the process exists, in part, in the degree to which the 
resulting improvera.ents can be sustained. The new 
performance monitoring system was key to sustaihing the 
improvements that had been made in the Captioned Films ^ 
Program.. 
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We held meetings with the di-rectorr branch chiefs and 
program staff of the *new division to explain the evaluation 

process that the program had been undergoing Xor ten 

« 

months. The EA and RFA Final Reports were reviewed and 
discussed, and compared with the new procurements which 
had , by that time , been is sued ♦ We then reviewed , piece by 
piece, the suggested performance monitoring system with the 
project officer newly responsible for the program. 

The nev^ division director and project officer decided 
to implement the performance monitoring system in whole. 
Copies of the prototype irtodels and performance monitoring 
.charts were distributed to alT SEP staff associated with 
the program. In addition, copies of the prototype models 
(which also described the evaluation questions, and hence 
the evaluative data to be, collected during each program 
activity) were d^tributed to the Captioned Films 
contractors to guide their operations. 
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CONCLUSIONS 

The evaluability assessm^t, rapid feedback 
assessment, and performartce monitoring system described in 
this paper took eleven months to complete. The informal:ion 
^produced during the assessment process was used, by SEP to 
underst;and the complexities of how the Captioned Films Loan 
Service for the Deaf actually functioned, and to change the 



program where, it was disf unctional . By ^ incorporating 
recommended program modifications into the new Captioned 
Films' procurements, SEP was able to go some, way toward 
ensuring that. the improvements made in program function 
would continue.-' By asking for, and then implementing a new 
performance monitoring system, SEP was able to go the rest 
of the way toward assuring that the Captioned Films Loan 
Service for the Deaf would function effectively, and that 
measurement data would be- collected regi l^rlyjr allowing 
program operations to be modifi-ed and sjpdated whenever* 
necessary. 

i 
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APPENDIX A 

Prototype Model: ECF Selection and Captioning 



NEEDS ASSESSMENT 




CKHA nn*\ Coiicmcror 
review .md ij);ree upon 

.HM««.iry foraa clenrnnce 



Contrai tor conducts -a 
neoda nSHcssnenC of "ECK 
UHcTH every ye.ir or 
every oiher year 



Oin tractor rcqiieHCs 
users' iieeds for filas 
in ttpeclflc currlculua 
nr^^,~frFiii!e levels, 
nnd Hpeciric riln 
titles oil nn onKod.f^ 
b.isia (in nmMenre 
respciiiue ccirda 



Conir^icror r<> Meets 
intormntion on user 
noodu Tron ( I loi 
dt'poal tory ia.in«i(>ers 



EVALUATION CONCERNS 



• IVies I fit- needs .irruss- 
iti-iil jvUlicr Iiir»ircuiion 

I roa n repre^seiii.iilve 

• Uoefi rlie noed8 asHeKS- 

HilK'lt inforiMi (on 
ll».il li'iHl'j^ Itself fii 
(U-vi<lopmi*nt of Viil Id 
enrrleulu|^ prior I ilos? 



Contrnetor detemlneu, on the 
l)«iHis of needs nssesHaent tn- 
formntioii, currlculuoprlorlty 
.irena for wliicli educational 
flliw* should be Cfyittoi\ed 



Contractor conpiJes and reviews 
the needs nnd evaluative In- 
foraatton col lecced frow 



audience response cards twice 
each year 



ContCflCLtor and CKMA 
annua J ly deiernlUe 
pri*irl ty areas in 
whirh fllma are lo 
htf r t ^qu este<l 



C^'MA and Contractor review 
and inrtirporate uiiHollrlted 
tiser roeommiMidat io*iiH wlicnj 
received ^ 



• How is the needs aBsesaacnt 
Information used to Renorate 
priority ari?nH? 

• Are audlonrc respoos^e card 
data cowplled and reviewitl 
iwive per year? 

• Are rcRpoiise card data deuion- 
Hirahly used to re-ova I mite 
current ECF» and lu Identify 
eurririihim noedB andJIIwH 
for assessment and evaluation? 



Prototype Model: ECF Selection and Captioning (continued) 



EVALUATION 



SELECTION 



(U>ncr«ic(«>r 
request R ( I tlea 
jnd (U*Krrl pC f ons 
of ov.iiinble 
f I Itis i II each 
priority nren 
frttn .1 1 1 known 
cdiicnc (onj I f 1 In 
producers every 



iwi> yi'ars 



'Gontrnotor 
ln( 1 1 A(f*8 
fnUow-iip 
procedures for 
a 1 1 prodiirers 
not resptiiultn^ 
CO* ri*qtii*sc for 
cit tos 



C«>nirnctor 
reviews al 1 
films snhfflltted 
an<t pel eels 
for screen iiif* ^ 
tl](»S(,* most 
t'losely relnted 
CO ilio curri- 
culum prior Utes 



At] films eliosen durJuK proseiec- 
I Ion nre reviewed In the field 



Field reviewers sulttal c their 
recomaiend«il Ions of films it) be 
c.ipcloiu'd 



Al I reromaonded M tni« are screened 
:i( a vnl ld.it lt»n workshop 



(Uuurnecur develops 1 Isl of 
— |— vrt-Hdiiled films, Uileh isail.ml iced 
CO CKHA 



CKHA uud Concracior 
review val I da led 
recomibendntions .iiid 
CFHA mnkca final 
selection of f J Jitts 



-< 



UVALUATION CONCERNS 
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How effort > vely 
in a fol low-up 
prticedore used 
to ensure that 
all film pro- 
ducers have 
respond4*d to 
clit' rcfjuesi for 
f I Inus? 



Are the fl In 
p rune lec clou 
er leer la 
spec I r I t*tt and 
used ronslii** 
teucly hy 
Com raccor 
lUnf f? 

llow many f 1 1ms 
are el Imlunced 
during .prai- 
se lert J oil? 



What 



(h(» 



nri» 

I'ostH of, pre 
se le('( tohV 



• Are (lie crl terln used toovniiiatc 
f I lt«:i for rapcloninK clear and 
conprehfiisl vc7 

• Are the erlterla used reliably 
and coiisl scene I y hy field 
reviewers? 

• 1 s the t I n I ii{; p ro y I ded Co 
reviewers adefiuato"? 

• What are ifie total and iinic costs 
of field evaluation? 

• Are the criteria used to He led 

i' I lun nl (he vnlhhuliui workshop 
used cciiiHlKtencly and reliably? 

• ilow Huiuy fllDiH nre accepted/ 
relecied? 

• What nre the costA of the 
validation worknliop: per film 
evaluated/per I 1 l,n accepted? 



• Are fiius caCcp.orl^sed 
Iv currlciiliiin prior 1 ty 
cliroii|;hout the evalu- 
ation and select J on 
process? 

• Ilow nuiuy curriculun^ 
priorities are 
addressed by the 
fllns ^elected? 

• What criteria are 
used com;ike (he final 
r I Ira select Ion? 

• Ilow many fllras nre 
rejected during (he 
final review? ' 
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Prototype Model: ECF Selection and Captioning (continued) 



CXPTIONIt'G 



I 



Contractor 'iie>;oi latus 
piircltnse ol fUws 



Oon tractor ne^otlatc^ 
film ci>m|>any dcvclop- 
jti'n( or s(u»tt llHtH 



('i>nt r.ii'tor reviews 
H|iotrlii>> lists 



(U)iitrartor mjiia^fs 
KCK rnpt lonln); 
workshop (s) where: 

• captioned Rcrlptf: 
nre written for 
e«U'h f i Im 

• 1-2 paj'.e Kttidy 
Kiiltlcs for nsG 
with (loaf are 
developed from 
eaeh film's 
exial Iii>> t'ommer*- 
t litl guide 



CFMA re V t ews ca p- 
t loned scripts ami 
study Kuldcs 



Upon receipt of 
Bi rlpts, film labs 
produce capt loned 
negative new sound 
track and the answer 
prints 



OKflA reviews 
answer prints 



the 



Cxintrai'tor detcntilnes 
optimal number of 
prints needed 



KiJm labs devejop 
final prints TinU"^ 
a variable number 
of copies, which 
fli'c del I vered to 
till' Contractor 



ECF are distributed 
to the depositories 



F VALUATION COrCERNS 



• Arc the nej^otlation 
Koidcl lues adequate 
for effective use' 
by the Contractor? 

• What criteria does 
the Contractor use 

to ensure the adequacy 
of the KCF spott Inj; 
lists? 



What Is (he time 
and dol Inr cos! of 
the capt loiii nK 
woi kshop? 

What are the rosin 
of the capt Ion 
writ in};/study 2*jiide 
componeiitH of the 
enti re workshop, 
|>er f I Im? 



• In what ways does 
CFHA ensure (jiial I ty 
film jirhits for 
the least cost? 

• Are cr I ter la for 

revlei^Mif study 
KUl.des specified? 

4 What proport Ion of 
study guides pass 
rev I ew? 



• What rr I ter I a are 
used to .ludj'.e the 
M^l/illty of the 
answer prints? 

• Wh/it method Is 
used to deteitQliii* 
the number of 
prints needed per 
film? 



• What method Is used 
to determine which 
fi lots /I'e dl8trlbiitc>d 
towhicli dcpiis 1 tor t es? 
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ECF SELECTION AND CAPTIONTNG: PERFORMANCE MONITORING CHART 



Oocutoen t 



Pur pose 



NuCils Assessment 
Report 



To document and describe 

the procedures and results ot 

the nfr«4^^^sessment process 

It should he used to answer 
these evaluation questions: 

.• Does the needs ussesnnxsnt 
gather Information from a 
representative sample ot 
users? 

• Does the needs assessment 
solicit Information that . . 
lends itself to development 
of valli^ curriculum prior- 
ities? ' c 

• Mow is tlie needs assessment 
information used to generate 
priority areas? 

• Are audience response caU\ 
data complied and^j^viewed 
regularly? 



Contents 



• NunJ)cr (Z) of ECF users 
surveyed and responding to 
assessment out of the total 
user population and by . 
user cntegory 

• Description of the proce- 
dures and crlterin used Co 
generate curriculum prior- 
ities from tlie assessment 
data 

• Results of the assessntcnt 

• Syntheals of the audience 
response card summaries 



Cliart specifying the 
source(s) of encli curricu- 
lum priority; needs assess- 
ment; audience response 
eards; luisolicited recoift- 
mciuhittonn; other 



Status (11/82) 



Currently done, 
except for sum- 
mitries by user 
category 



Available from 
several docu- 
ments/Not orgnn- 
Ized 



Currently done 

Raw daca col- ^ 
lected/Hpt, ^ 
currently sum- 
marized ^ 

Not currently 
done 



Si>ecial Use 



Delivery 



Annual: to be 
delivered and 
reviewed during 
Joint contractor/ 
CFMA meeting prior 
to film evaluation 



Film Evaluation 
Keport 



3o 



ERIC 



The report should be used to 
answer these evaluatlor^ 
quest ions : 

• How effectively is a follow- 
up procedure used to ensure 
that all film producers have 
responded to the request for 
films? 

• Arc the film preselection 
eriterla specified, Md used 
consistently by Contractor 
staff? 

• How many films are eliminated 
during preselection? 

• Wliat are the costs of pre- 
selection? 

• Are the criteria used to 
evaluate films for caption- 
ing clear nnd comprehensive? 

• Are the criteria used reli- 
ably and Consistently by 

f leld- reviewers^' - 

• Is the training provided to 
reviewers adequate? 



• Number of film producers con- 
tacted; number (Z) respond- 
ing; numberof follow-up 
requests senrto those not 
responding; nun)I)ur {%) re- 
spoiullng to foUow-up 

• Criteria used during pre- 
selection 



• Sumnuiry of training provided 
to field eval gators 

• CrTteria used to evaluate 
films 

• I.is't of flints evaluated In 
the field (grouped hy prior- 
ity area) with the ratings 
trotn.eacli field site 

• NuRiber (2) of films recom- 
mended for validation; 
nunater (Z) re)t'elecL 

• CritorlTi^flod' TO T(£ylew 
films nt validation workshop 



Most Information 
currently avail- 
able from 
several sources/ 
Not documented 
or organized 

Exist/Not 
currently docu- 
mented 

Currently avnll- 
nble 

Currently avail- 
able 

Most informiition 
la currently 
available/Not 
reported as 
requested here 

Currently avnll- 
ahle 



ChrreiitTy avaiV- 
able 



Tlie film evaluation report 
should enable CKMA to monitor 
and document the effectiveness 
and accuracy of the evaluation 
process. It should be used to 
update and assess the relative 
merits and casta of eacli step 
of the process.. 



Annual 
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ECF SELECTION AND CAPTIONING: PlilRFORMANCE MONITORING CHART - continued 



I)iicumu4U 



Purpose 



ContentB 



Status (11/82) ' 



Film Evaluation 
Kuport continued 



• Wliat are the total and unit 
costs of field evaluation? 



Llbt of films reviewed at 
validation workshop (grouped 
by priority area) with each 
review group's rating and 
contractor's sntninary rating 

Number (Z) of films reconj- 
mcnded/Not recommended for 
captioning 

Uitdget breakdown for entire 
evaluation process, detailij 
ing! 

1 - time and dollar costs 
for preselection 
activity 

2 -costs of field evaluation 



Most inforiitatlon 
is currently 
available /Not 
reported as 
requested here 

Currently avail- 
able 

Moat information 
--Is currently 
nvailable/Not 
reported as 
requested here 



Special Use 



Delivery 



Film Selection 
Report . 



> 
I 



To document the final film 
selection process 

Tliesc questions should be 
answered: 

• Are films categorized by 
curriculum p.riorlties 
throughout the process? 

• itow many curriclum prior- 
ities are addressed by the 
films selected? 

• Uow many films were 
rejected during the final 
review? 

• Wl^at criteria are used to 
make the final film 
selecti^iTn? 



List of films selected from 
recommendations (grouped by 
curr^ulum priority) 

Criteria used for final 
selection 

1,ist of films rejected 
during final selection, with 
reason given for earh 

hist of curriculum prior- 
ities not met by sele'cted 
films 



Information 
available/Not 
organized as 
requested here 

Exist/Not 
documented 

Exists/Not 
documented 

In formation 
available/Not 
currently 
reported 



Anflual - incorpor 
a ted" Into the 
film evaluation 
report 



Captioning 
Report 



ERIC 



To answer these questions: 

• Are the negotiation^gnide- 
lines adequate for effec- 
tive use by the^ contractor? 

^ 

• Wliat criteria does' the 
contractor use to ensure 
the adequacy of Che KCP 
spotting lists? 

• Wliat Is ,thii, time and dollar 
ccist of thd captioning 
workshop? 



• Criteria used to review 
spotting 1 IstH 

• Budget breakdown for cap- 
Cloning workshop, 
including: 

I - total costs 

2* costs of caption writing 
as^ % of total and per 
film 

.3- costs of study guide 
writing as t of t'otal 

.„ . and. |i(ir, film 

A - t Una • requl red' tor wark- 
Hhiip 



Kxlst/Not 
documented 

Currently avail- 
able 



Annual 



EOF SELECTION AND CAPTIONING: PERFORMANCE MONITORING CHART - continued 



Docunent 



I'urpoae 



Contcnly 



Stotun (11/82) 



Spec In 1 Use 



Del 1 very 



CnpC inning 

Report - cnnclnued 



In what ways docs CFMA ensure 
quality fllo prints fnr the 
least cost? 

Are criteria for review of 
stitdy guides (Lesson Guides) 
specified? 



• Criteria used to Judge qual- Not docuacnCud 
it^ of andVer prints 

t Hetlind used to deteralne Currently . 

ntiolter of prints needed per available 
flJii with list of nunber of 
prints ordered 



/ 



• Wliat proportion of study 
guides i>a9s review? 

• Wliat criteria are used to 
judge the quality of the 
answer prints? 

• V}iat aethod is ttscd to deter- 
' atne the nunber of prints 

needed per. f lloi? 

• Wliat ecthod is used to deter- 
mine which files are distri- 
buted to wlilch depositories? 




I 

Ok 



4vj 
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